Sticky-Finger Manipulation With A Multi-Touch Interface

Manipulation of complex simulated entities like cloth in graphical applications can be non-intuitive and awkward for the user due to the high number of degrees of freedom in their motions. In robotics, direct control of a robotic manipulator for a highlydexterous task is similarly tedious because multiple joints have to be controlled simultaneously, in close coordination and often in varying velocities. Conventional interaction devices used in these applications, such as mice and joysticks, fail to provide a natural mapping from their relatively simple command space to the high-dimensional task space associated with the entity being controlled. In our paper, we present a userintuitive system for the direct control of such entities using a multi-touch interface. The user manipulates simulated or real-world objects in a natural fashion by placing his or her fingers at desired points of the target objects and moving these fingers over the multi-touch surface. Any finger that is directly in contact with part of a target object becomes sticky, causing any underlying object to translate along with that finger. Our sticky finger technique allows the user to intuitively produce a range of complex manipulations such as pinching, folding, draping and tearing using common multi-touch gestures. We demonstrate our user interface in two applications, namely real-time cloth manipulation in a graphical application and the teleoperation of a human-like robot hand for common dexterous multi-fingered tasks.